Pupil modulation as a cognitive control signal

ABSTRACT

One exemplary implementation provides an improved user experience on a device by using physiological data to initiate a user interaction for the user experience based on an identified interest or intention of a user. For example, a sensor may obtain physiological data (e.g., pupil diameter) of a user during a user experience in which content is displayed on a display. The physiological data varies over time during the user experience and a pattern is detected. The detected pattern is used to identify an interest of the user in the content or an intention of the user regarding the content. The user interaction is then initiated based on the identified interest or the identified intention.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of U.S. patent application Ser. No.16/568,782, filed Sep. 12, 2019, which claims the benefit of U.S.Provisional Application Ser. No. 62/738,031 filed Sep. 28, 2018, each ofwhich is incorporated herein in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to providing user experienceson electronic devices, and in particular, to systems, methods, anddevices for using physiological data to improve a user experience.

BACKGROUND

Electronic devices have different capabilities with respect to viewingand interacting with electronic content. A variety of input mechanismshave been incorporated into a variety of user devices to providefunctionality and user interaction (e.g., keyboards, mice, touchscreens,buttons, microphones for voice commands, optical sensors, etc.). Forexample, touch screens have been incorporated into mobile phones (e.g.,smartphones), tablet computers, wearable devices (e.g., watches,glasses, head-mounted devices, etc.), and other computing devices,allowing software developers to create engaging software applications(“apps”) for entertainment, productivity, health, and the like. In someinstances, touch screens work in conjunction with a variety of otherinput mechanisms for interacting with a device (e.g., optical sensors,buttons, microphones for voice commands, etc.).

Many devices, however, can have limited device interaction and controlcapabilities due to device size constraints, display size constraints,operational constraints, etc. For example, small or thin user devicescan have a limited number of physical buttons for receiving user input.Similarly, small user devices can have touchscreens with limited spacefor providing virtual buttons or other virtual user interface elements.In addition, some devices can have buttons or other interactive elementsthat are unnatural, cumbersome, or uncomfortable to use in certainpositions or in certain operating conditions. For example, it may becumbersome to interact with a device using both hands (e.g., holding adevice in one hand while engaging interface elements with the other). Inanother example, it may be difficult to press small buttons or engagetouchscreen functions while a user's hands are otherwise occupied orunavailable (e.g., when wearing gloves, carrying groceries, holding achild's hand, driving, etc.). In still other examples, deviceinteraction can be limited in a variety of other ways.

SUMMARY

Various implementations disclosed herein include devices, systems, andmethods that obtain physiological data (e.g., pupil dilation,electroencephalography, etc.) of a user during a user experience, inwhich content is displayed on a display, with one or more physiologicalsensors. The physiological data varies over time during the userexperience and a pattern is detected. Moreover, in some implementations,the physiological data includes involuntary user responses. Dependentupon user privacy or opt-in/out settings, the detected pattern may beused to help or assist the user by identifying an interest of the userin the content or an intention of the user regarding the content.Identifying the intention may include, for example, identifying anintent to execute a movement, make a decision, or select a target in thecontent at a particular instant in time or in the future. Thus, a userinteraction may be initiated based on the identified interest or theidentified intention. In some implementations, the detected pattern isunique to the user and is stored in a user profile associated with theuser. For example, the user profile may be used to provide apersonalized user experience that identifies the user's interest orintention based upon the user's unique detected patterns.

In some implementations, the physiological data is pupil dilation dataand represents a time-varying pupil diameter. Thus, the detected patternmay be a pupil dilation pattern. In some implementations, exogenoussignals are accounted for when detecting the pupil dilation pattern. Forexample, exogenous signals may result from ambient light changes,chromatic changes, accommodation of the eye, content lighting changes,cyclical pupil dilations, changes in ambient noise, or changes in motionof the device. In some implementations, a machine learning technique istrained to identify patterns in physiological data corresponding to userinterests or user intentions.

In some implementations, additional data is obtained, and the interestor intention is identified based on that data. The data may include, forexample, a gesture of a body part detected by an image sensor during theuser experience, a voice command of a voice detected by a sound sensorduring the user experience, a fixed gaze detected by an eye sensorduring the user experience, a sequence of gaze patterns detected by aneye sensor during the user experience, an orienting response (e.g., headmovement), a movement detected by a motion sensor during the userexperience, a facial expression detected by an image sensor during theuser experience, or an attribute included in the content. In someimplementations, the method includes identifying related interests orintentions based on previously identified interests or previouslyidentified intentions during the user experience. In someimplementations, the method includes determining a confidence in theidentified interest or the identified intention based on previouslyidentified interests or previously identified intentions during the userexperience. In some implementations, data is received regarding avoluntary user movement (e.g., an arm movement) and the voluntary usermovement is interpreted as an intention to interact with the contentbased on an involuntary characteristic of the user that is captured inthe physiological data. For example, the user can use natural armgestures and the gestures will only be recognized as intentionalcommands when the involuntary changes of the user's pupil reveal thatintention. Moreover, identifying the interest may include identifying aninterest in a particular object in content at a particular instant intime or in the future.

In some implementations progressive interfaces aid the usability ofinteractions supported by an involuntary characteristic of the user thatis captured in the physiological data. For example, system feedback tothe user may explicitly call out the use of a lower-confidencemultimodal signal to begin a low-commitment interaction with someinitial feedback to the user (e.g., highlighting or selecting an object,or displaying one or more menu items). Further input from the user inresponse to the low-commitment interaction may progressively lead to ahigher-commitment action (e.g., acting on or deleting an item).

In some implementations initiating the user interaction includesproviding additional content association with an object in the content,e.g., additional content corresponding to the identified interest or theidentified intention. In some implementations, initiating the userinteraction includes removing an object in the content based on theidentified interest or the identified intention. In someimplementations, initiating the user interaction includes automaticallycapturing images of the content at times during the user experiencedetermined based on the identified interest or the identified intention.

In some implementations, detecting the pattern includes tracking aphysiological attribute associated with the physiological data using afirst sensor and activating a second sensor to obtain the physiologicaldata based on the tracking. In some implementations, the device, e.g., ahead-mounted-device, handheld device, laptop, or desktop, utilizesonboard sensors to obtain the physiological data and, in otherimplementations, the device includes a combination of multiple andphysically separate devices and sensors.

In some implementations, informed consent is received from the user toobtain the physiological data and/or additional data. Moreover, a usermay consent, opt-in/out to feature benefits, or select certain actionsor types of action that may be invoked automatically based onphysiological data. In some implementations, a prompt to consentincludes a graphical cue rendered on a display portion of the device.For example, a graphical user interface may include a request forpermission and an input portion that allows the user to input consent.In some implementations, a prompt to consent includes an audio cuegenerated. For example, the audio cue may include an audio segmentexplanatory of a request for permission and the consent may be receivedwhen the user issues a voice response that indicates consent.

In accordance with some implementations, a device includes one or moreprocessors, a non-transitory memory, and one or more programs; the oneor more programs are stored in the non-transitory memory and configuredto be executed by the one or more processors and the one or moreprograms include instructions for performing or causing performance ofany of the methods described herein. In accordance with someimplementations, a non-transitory computer readable storage medium hasstored therein instructions, which, when executed by one or moreprocessors of a device, cause the device to perform or cause performanceof any of the methods described herein. In accordance with someimplementations, a device includes: one or more processors, anon-transitory memory, an image sensor, and means for performing orcausing performance of any of the methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood by those of ordinaryskill in the art, a more detailed description may be had by reference toaspects of some illustrative implementations, some of which are shown inthe accompanying drawings.

FIG. 1 illustrates a device displaying content and obtainingphysiological data from a user in accordance with some implementations.

FIG. 2 illustrates a pupil of the user of FIG. 1 in which the diameterof the pupil varies with time.

FIG. 3A is a chart illustrating detection of a pattern of physiologicaldata in accordance with some implementations.

FIG. 3B is a chart illustrating physiological data and an exogenoussignal in accordance with some implementations.

FIG. 4 is a block diagram illustrating device components of an exemplarydevice according to some implementations.

FIG. 5 is a block diagram of an example head-mounted device (HMD) inaccordance with some implementations.

FIG. 6 is a flowchart representation of a method for initiating a userinteraction based on an identified interest or an identified intentionof a user where the interest or the intention is identified based onphysiological data of the user obtained using a sensor during a userexperience.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

DESCRIPTION

Numerous details are described in order to provide a thoroughunderstanding of the example implementations shown in the drawings.However, the drawings merely show some example aspects of the presentdisclosure and are therefore not to be considered limiting. Those ofordinary skill in the art will appreciate that other effective aspectsor variants do not include all of the specific details described herein.Moreover, well-known systems, methods, components, devices and circuitshave not been described in exhaustive detail so as not to obscure morepertinent aspects of the example implementations described herein.

FIG. 1 illustrates a device 10 displaying content 15 and obtainingphysiological data 45 from a user 25. While this example and otherexamples discussed herein illustrate a single device 10 in a real-worldenvironment 5, the techniques disclosed herein are applicable tomultiple devices as well as to other real-world environments. Forexample, the functions of device 10 may be performed by multipledevices. Furthermore, this example and other examples discussed hereinoperate under the assumption that the user 25 has provided informedconsent to benefit from the techniques disclosed herein. For example,the user 25 may consent to the tracking or use of physiological data 45,opt-in/out to feature benefits, or select certain actions or types ofaction that may be invoked automatically by device 10 based onphysiological data 45.

In some implementations, as illustrated in FIG. 1, the device 10 is ahandheld electronic device (e.g., a smartphone or a tablet). In someimplementations the device 10 is a laptop computer or a desktopcomputer. In some implementations, the device 10 has a touchpad and, insome implementations, the device 10 has a touch-sensitive display (alsoknown as a “touch screen” or “touch screen display”). In someimplementations, the device 10 is a wearable head mounted display(“HMD”).

In some implementations, the device 10 includes an eye tracking systemfor detecting eye position and eye movements. For example, an eyetracking system of an HMD may include one or more infrared (“IR”)light-emitting diodes (“LEDs”), an eye tracking camera (e.g., near-IR(“NIR”) camera), and an illumination source (e.g., an NIR light source)that emits light (e.g., NIR light) towards the eyes of the user 25.Moreover, the illumination source of the HMD may emit NIR light toilluminate the eyes of the user 25 and the NIR camera may capture imagesof the eyes of the user 25. In some implementations, images captured bythe eye tracking system may be analyzed to detect position and movementsof the eyes of the user 25, or to detect other information about theeyes such as pupil dilation. For example, the point of gaze estimatedfrom the eye tracking images may enable gaze-based interaction withcontent shown on the near-eye display of the HMD.

In some implementations, the device 10 has a graphical user interface(“GUI”), one or more processors, memory and one or more modules,programs or sets of instructions stored in the memory for performingmultiple functions. In some implementations, the user 25 interacts withthe GUI through finger contacts and gestures on the touch-sensitivesurface. In some implementations, the functions include image editing,drawing, presenting, word processing, website creating, disk authoring,spreadsheet making, game playing, telephoning, video conferencing,e-mailing, instant messaging, workout support, digital photographing,digital videoing, web browsing, digital music playing, and/or digitalvideo playing. Executable instructions for performing these functionsmay be included in a computer readable storage medium or other computerprogram product configured for execution by one or more processors.

In some implementations, the device 10 presents an experience in whichcontent 15 is displayed on a display of the device 10 during a userexperience. A sensor 20 detects physiological data 45 of the user 25during the user experience. In some implementations, the device 10employs various physiological sensor, detection, or measurement systems.Detected physiological data may include, but is not limited to,electroencephalography (EEG), electrocardiography (ECG),electromyography (EMG), functional near infrared spectroscopy signal(fNIRS), blood pressure, skin conductance, or pupillary response.Moreover, the device 10 may simultaneously detect multiple forms ofphysiological data 45 in order to benefit from synchronous acquisitionof physiological data 45. Moreover, in some implementations, thephysiological data 45 represents involuntary data, i.e., responses thatare not under conscious control. For example, a pupillary response mayrepresent an involuntary movement.

In some implementations, one or both eyes 30 of the user 25, includingone or both pupils 35 of the user 25 present physiological data 45 inthe form of a pupillary response. The pupillary response of the user 25results in a varying of the size or diameter of the pupil 35, via theoptic and oculomotor cranial nerve. For example, the pupillary responsemay include a constriction response (miosis), i.e., a narrowing of thepupil, or a dilation response (mydriasis), i.e., a widening of thepupil. In some implementations, the device 10 may detect patterns ofphysiological data 45 representing a time-varying pupil diameter.

FIG. 2 illustrates a pupil 35 of the user 25 of FIG. 1 in which thediameter of the pupil 35 varies with time. As shown in FIG. 2, a presentphysiological state 50 may vary in contrast to a past physiologicalstate 55. For example, the present physiological state may include apresent pupil diameter and a past physiological state may include a pastpupil diameter.

The physiological data 45 may vary in time and the device 10 may use thephysiological data 45 to detect a pattern. In some implementations, thepattern is a change in physiological data 45 from one time to anothertime, and, in some other implementations, the pattern is series ofchanges in physiological data over a period of time. Based on detectingthe pattern, the device 10 may assist user 25 by identifying an interestor intent 40 of the user 25 and may initiate a user interaction based onthe identified interest or intent 40.

FIG. 3A is a chart 300 illustrating detection of a pattern ofphysiological data 45. Chart 300 illustrates a time-varying pattern 305of physiological data 45, for example, an amount of pupil dilation(y-axis) over time (x-axis). The pattern 305 includes a peak pattern 310that may be interpreted by device 10 as an indication of an interest orintent 40 of the user 25. In some implementations, the device 10utilizes a model trained to determine that the peak pattern 310indicates that the user 20 is involuntarily signaling interest or intent40 during the time of the peak pattern 310, e.g., based on something theuser is looking at or otherwise experiencing during that time period. Insome implementations, a machine learning model (e.g., a trained neuralnetwork) is applied to identify patterns in physiological data 45.Moreover, the machine learning model may be used to match the patternswith learned patterns corresponding to indications of interest or intent40. The device 10 may learn patterns specific to the particular user 25.For example, the device 10 may learn from determining that peak pattern310 represents an indication of interest or intent of the user 25 anduse this information to adapt the model to subsequently identify thesimilar peak pattern 320 as another indication of interest or intent ofthe user 25. Such learning can take into account the user's interactionsthat may confirm predictions of interest or intent, e.g., if the modelpredicts an intent to click a button and the user clicks the button, themodel can be updated accordingly. As another example, the model may betrained or adapted based on the user imagining pressing a button, i.e.,not physically pressing the button.

FIG. 3B is a chart 350 illustrating physiological data 45 (e.g., basedon measured pupil dilation) and an exogenous signal 360 (e.g., a measureof ambient light at the device 10). The exogenous signal 360 correspondsto a measure of any factor or factors that could influence thephysiological data. For example, the amount of ambient light at thedevice 10 may result in changes to the amount of dilation of the user'seyes, e.g., a decrease in ambient light may result in more dilation ofthe eyes, etc.

In some implementations, the physiological data 45 is adjusted orotherwise interpreted based on the exogenous signal 360. For example,the peak pattern 310 corresponding to a dilation of the user's eyes maybe preliminarily interpreted as an indication of intent or interest.Since the exogenous data 360 is level during the time period of peak 310(e.g., indicating a constant ambient light level), the determinationthat the dilation should be interpreted as an indication of intent orinterest is accepted.

In contrast, the peak pattern 390 corresponding to a dilation of theuser's eyes may similarly be preliminarily interpreted as an indicationof intent or interest but this determination may be rejected. In thisexample, the exogenous signal 360 indicates an increase 370 followed bya decrease 380 during the same time period as the peak pattern 390.Thus, the exogenous signal could correspond to an exogenous factor(rather than an interest or intent of the user 25) that caused the peakpattern 390. Accordingly, the device 10 may reject the preliminarilyinterpretation of peak pattern 390 as an indication of intent orinterest. In some implementations, a model is used to account forexogenous signals, e.g., the model is trained to interpret patterns inphysiological data 45 that occur during the same time periods aspatterns in exogenous signals.

In some implementations, exogenous signals corresponding to pupildiameter changes in the pupil dilation data result from ambient lightchanges, chromatic changes, accommodation of the eye, content lightingchanges, cyclical pupil dilations, a change in ambient noise, or changein motion of the device. For example, an increase in ambient light maycorrespond to a decreased pupil diameter. Likewise, chromatic changes orchanges in content lighting may correspond to increases or decreases inpupil diameter. Moreover, exogenous signals may also be related to theuse of drugs or medications. For example, opiates or opioids may beassociated with a narrowing of the pupil and cocaine or amphetamines maybe associated with a widening of the pupil. In some implementations,based on pupillary response, the device 10 detects patterns thatcorrespond to one or more exogenous factors, for example, based onlearning such patterns using a machine learning model. The device 10 maydistinguish patterns of physiological data 45 that correspond tointerest or intent from patterns of physiological data 45 thatcorrespond to exogenous factors, for example, based on learning suchdistinctions using a machine learning model.

In some implementations, the device 10 detects the location of the eyes30 of the user 25 and the pupils 35 of the user 25, e.g., by processingand analyzing an image comprising light (typically infrared and/or acolor produced by the red-green-blue additive color model) reflectingfrom one or both eyes, in order to locate and measure a diameter of thepupils. The reflected light may originate from a light projecting sourceof the device 10, or any other natural (e.g., sunlight) or artificial(e.g., a lamp) source. Using techniques such as detecting pupil centerand corneal reflections (PCCR), the device 10 may process and analyze animage comprising light reflecting from an element of the eye 30,including the pupil 35, in order to determine the diameter of the pupil35. Additionally, the device 10 may process light (e.g., from anillumination source on the device or elsewhere) reflected off the eye 30of the user 25 as a glint.

In some implementations, the location and features of the head 27 of theuser 25 (e.g., an edge of the eye, a nose or a nostril) are extracted bythe device 10 and used in finding coarse location coordinates of theeyes 30 of the user 25, thus simplifying the determination of preciseeye 30 features (e.g., position, gaze direction, etc.) and making thepupil diameter 50 measurement more reliable and robust. Furthermore, thedevice 10 may readily combine the 3D location of parts of the head 27with gaze angle information obtained via eye part image analysis inorder to identify a given on-screen object at which the user 25 islooking at any given time. In some implementations, the use of 3Dmapping in conjunction with gaze tracking allows the user 25 to movetheir head 27 and eyes 30 freely while reducing or eliminating the needto actively track the head 27 using sensors or emitters on the head 27.

By tracking the eyes 30, some implementations reduce the need tore-calibrate the user 25 after the user 25 moves their head 27. In someimplementations, the device 10 uses depth information to track thepupil's 35 movement, thereby enabling a reliable pupil diameter 50 to becalculated based on a single calibration of user 25. Utilizingtechniques such as pupil-center-corneal reflection (PCCR), pupiltracking, and pupil shape, the device 10 may calculate the pupildiameter 50, as well as a gaze angle of the eye 30 from a fixed point ofthe head 27, and use the location information of the head 27 in order tore-calculate the gaze angle. In addition to reduced recalibrations,further benefits of tracking the head 27 may include reducing the numberof light projecting sources and reducing the number of cameras used totrack the eye 30.

Some implementations provide the device 10 with faster, more efficientmethods and interfaces for navigating through user interfaces. Suchmethods and interfaces may complement or replace conventional methodsfor navigating through user interfaces. Such methods and interfaces mayreduce the cognitive burden on a user 25 and produce a more efficienthuman-machine interface. For battery-operated computing devices, suchmethods and interfaces may conserve power and increase the time betweenbattery charges. Moreover, some implementations enhance the navigationof user interfaces based on detecting patterns associated withphysiological data 45.

In accordance with some implementations, a user interface having one ormore selectable objects is displayed on a screen of the device 10 andthe interest or intent 40 of the user 25 is associated with one of theselectable objects. Moreover, in some implementations, the interest orintent 40 of the user 25 is associated with selecting one of the one ofthe selectable objects. In some implementations, the device 10 receivesan input that corresponds to a first gesture and confirms the input withthe identified interest or intent 40 of the user 25. In someimplementations, the first gesture is received by the device 10 asvoluntary data, i.e., behavior over which the user 25 has control. Forexample, voluntary data may be received based on the user's voiceinputs, hand gestures, touch input, keystrokes, etc. In someimplementations, the interest or intent 40 of the user 25 is associatedwith multiple types of input (i.e., multimodal) communicated with thedevice 10 by the user 25. For example, more than one low-commitmentvoluntary interactions may, in combination, be associated with theinterest or intent 40 of the user 25.

In some implementations, in response to receiving an input thatcorresponds to the identified interest or intent 40, the device 10searches for a target selectable object and move or otherwise alter anobject selection indicator. When a target selectable object isidentified, the device 10 may move an object selection indicator towardsthe target selectable object. When a target selectable object is notidentified, moving the object selection indicator may include moving anobject selection indicator in accordance with the identified userinterest or intent 40.

In some implementations, the device 10 searches for the targetselectable object based on the identified user interest or intent 40. Insome implementations, the device 10 moves an object selection indicatorin accordance with the user interest or intent 40 by calculating atrajectory of the object selection indicator based on a first identifieduser interest or intent 40 and then terminate the trajectory of theobject selection indicator based on a second identified interest orintent 40. Moreover, when calculating a trajectory of the objectselection indicator based on the identified user interest or intent 40,the device 10 may search for one or more candidate selectable objectsthat meet predefined candidate criteria, and when one or more candidateselectable objects are found, the device 10 may identify a respectivecandidate selectable object as the target selectable object. Forexample, the candidate selectable object may be identified as the targetselectable object based on proximity of the candidate selectable objectto a termination point of the trajectory (e.g., selectable object isclosest to or within a predefined distance of a termination point of thetrajectory).

In some implementations, when no candidate selectable objects are found,the device 10 moves the object selection indicator in accordance withthe identified user interest or intent 40 to a termination point of thetrajectory. In some implementations, the trajectory is calculated basedon simulated physical properties of the object selection indicator. Insome implementations, the object selection indicator is not visible tothe user 25.

Computing devices are provided with faster, more efficient methods andinterfaces for navigating through user interfaces, thereby increasingthe effectiveness, efficiency, and user satisfaction with such devices.Such methods and interfaces may complement or replace conventionalmethods for navigating through user interfaces.

In some implementations, the device 10 uses a detected pattern ofphysiological data to control a function of the device 10. In someimplementations, the device 10 identifies a given interactive itempresented on a display of the device 10 at a point of the interest 40 ofthe user 25 (e.g. at a position in the direction of the user's gaze) andchanges a state of the given interactive item responsively based on theidentified interest or intent 40 of the user 25.

In some implementations, changing the state of the given interactiveitem includes performing an operation associated with the giveninteractive item. For example, interactive items may include menuchoices that the user 25 can select to present specific content (e.g., amovie or a television show) on the display 15. In some implementations,the device 10 changes the state of a given interactive item by directinginput received from the user 25 to the given interactive item. In someimplementations, the device 10 identifies the given interactive itembased on other data and interact with the interactive item based on theidentified user interest or intent 40. For example, if the user 25 isgazing at a selectable button, the device 10 may identify the selectablebutton based on the user's gaze and then select the selectable buttonbased on the identified user interest or intent 40.

In some implementations, the device 10 identifies a given interactiveitem presented on the display of the device 10 at a position in thedirection of the user's gaze. Moreover, the device 10 may change a stateof the given interactive item responsively to a spoken verbal commandreceived from the user 25 in combination with the identified interest orintent 40 of the user 25. For example, the given interactive item maycomprise an icon associated with a software application, and the user 25may gaze at the icon and say the word “start” to execute theapplication. The device 10 may then use the identified interest orintent 40 of the user 25 as a confirmation of the user's verbal command.In some implementations, the device 10 is configured to identify a giveninteractive item responsively to the direction of the user's gaze, andto manipulate the given interactive item responsively to a gestureperformed by a limb or body part (e.g., a finger or a hand). The device10 may then confirm the gesture based on identifying user interest orintent 40. In some implementations, the device 10 removes an interactiveitem or object based on the identified interest or intent 40. In otherimplementations, the device 10 automatically captures images of thecontent at times when the interest or intent 40 of the user 25 isdetermined.

In some implementations, the device 10 is configured to provide aprogressive interface. The progressive interface may aid the usabilityof voluntary inputs from the user 25 in combination with, or supportedby, the physiological data 45 (e.g., one or more involuntarycharacteristics of the user 25). For example, the device 10 may provideprogressive feedback to the user 25 regarding an identified interest orintent 40 of the user 25. In some implementations, the device 10 beginsa low-commitment interaction with the user 25 in response to detecting apattern of physiological data or in response to receiving voluntary userinput. For example, in response to one or more lower-confidencedetections of user interest or intent 40, the device 10 may perform alow-commitment interaction (e.g., highlighting or selecting an object,or displaying one or more menu items) with the user 25. In someimplementations, the low-commitment interaction may direct the user 25to progressively perform higher commitment actions to confirm the userinterest or intent 40. For example, the device 10 may act on or deletean item in response to further input from the user 25.

As a power saving feature, the device 10 may detect when the user 25 isnot looking at the display and the device 10 may activate power savingtechniques, e.g., disabling physiological sensors when the user 25 looksaway for more than some threshold period of time. Furthermore, in someimplementations, the device 10 dims or darkens the display (i.e.,decrease the brightness) entirely when the user 25 is not looking at thedisplay. When the user 25 looks back toward the display, the device 10may deactivate the power saving techniques. In some implementations, thedevice 10 tracks a physiological attribute using a first sensor and thenactivates a second sensor to obtain the physiological data 45 based onthe tracking. For example, the device 10 may use a camera to identifythat the user 25 is looking in the direction of the device 10 and thenactivate an eye sensor when it is determined that the user 25 is lookingin the direction of the device 10.

In some implementations, a combination of determining user intent and aninput device 10 is used to create an interactive user interface thatutilizes the input device 10 to identify an on-screen interactive itemand determine the user's interest or intent 40 in interacting with theon-screen interactive item. For example, a user 25 may use a mouse toselect an on-screen interactive item based on the user's determinedinterest or intent 40, e.g., a “mouse click” type event triggered bydetermining the user's interest or intent 40 rather than a mouse click.In some implementations, a combination of determining user intent andgaze tracking is used to create an interactive user interface that candetect which on-screen interactive item the user 25 is looking at (e.g.,gaze tracking) and determine the user's interest or intent 40 ininteracting with the on-screen interactive item, thereby obviating theneed for a mouse and/or a keyboard.

Furthermore, the combination of determining user interest or intent 40with other modalities, such as gaze tracking, facial gesture detection,3D mapping/gesture detection and/or voice detection, enables the user 25to control on-screen objects fully, without the use of a mouse or atouch screen. In this manner, the user 25 can perform a full range ofpointing and selection functions, including searching through largenumbers of information items and choices. The combined interfacemodalities may also be used to search and perform control functionswithin the context of a certain interactive item, such as performingfind, cut, copy and paste functions within an open file. In someimplementations, the device 10 identifies a first interest, or group offirst interests, and then progressively identifies a second interest, orsecond group of interests, based on the previously identifiedinterest(s).

When selecting a given interactive item, the device 10 may convey visualfeedback or audio feedback to the user 25 indicating the selection(i.e., before performing an action such as presenting selectinginteractive content). Examples of visual feedback include changing thesize and/or appearance of the selected item or highlighting the selecteditem by surrounding the selected item with a border. Examples of audiofeedback include non-verbal types of audio feedback (e.g., clicks,beeps, chirps, or other various types of non-verbal sound effects) aswell as audio feedback with varying levels of verbosity (e.g.,outputting one or several spoken words confirming a selection). In someimplementations, conveying visual or audio feedback enhances the user 25experience by signaling the user 25 that an action is being taken, orwill be taken, based on the user's interest or intent 40.

In some implementations, the device 10 utilizes a training orcalibration sequence to adapt to the specific physiologicalcharacteristics of a particular user 25. In some implementations, thedevice 10 presents the user 25 with a training scenario in which theuser 25 is instructed to interact with on-screen items. By providing theuser 25 with a known intent or area of interest (e.g., viainstructions), the device 10 may record the user's physiological data 45and identify a pattern associated with the user's intent or interest 40.For example, the device 10 could direct a user to mentally select thebutton in the center of the screen on the count of three and record theuser's physiological data 45 to identify a pattern associated with theuser's intent or interest 40. In some implementations, the patternassociated with the user's intent or interest 40 is stored in a userprofile associated with the user and the user profile can be updated orrecalibrated at any time in the future. For example, the user profilecould automatically be modified over time during a user experience toprovide a more personalized user experience.

FIG. 4 is a block diagram of an example of a device 10 in accordancewith some implementations. While certain specific features areillustrated, those skilled in the art will appreciate from the presentdisclosure that various other features have not been illustrated for thesake of brevity, and so as not to obscure more pertinent aspects of theimplementations disclosed herein. To that end, as a non-limitingexample, in some implementations the device 10 includes one or moreprocessing units 402 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs,processing cores, and/or the like), one or more input/output (I/O)devices and sensors 406, one or more communication interfaces 408 (e.g.,USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x,GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and/or the liketype interface), one or more programming (e.g., 110) interfaces 410, oneor more displays 412, one or more interior and/or exterior facing imagesensor systems 414, a memory 420, and one or more communication buses404 for interconnecting these and various other components.

In some implementations, the one or more communication buses 404 includecircuitry that interconnects and controls communications between systemcomponents. In some implementations, the one or more I/O devices andsensors 406 include at least one of an inertial measurement unit (IMU),an accelerometer, a magnetometer, a gyroscope, a thermometer, one ormore physiological sensors (e.g., blood pressure monitor, heart ratemonitor, blood oxygen sensor, blood glucose sensor, etc.), one or moremicrophones, one or more speakers, a haptics engine, one or more depthsensors (e.g., a structured light, a time-of-flight, or the like),and/or the like.

In some implementations, the one or more displays 412 are configured topresent a user experience to the user 25. In some implementations, theone or more displays 412 correspond to holographic, digital lightprocessing (DLP), liquid-crystal display (LCD), liquid-crystal onsilicon (LCoS), organic light-emitting field-effect transitory (OLET),organic light-emitting diode (OLED), surface-conduction electron-emitterdisplay (SED), field-emission display (FED), quantum-dot light-emittingdiode (QD-LED), micro-electro≙mechanical system (MEMS), a retinalprojection system, and/or the like display types. In someimplementations, the one or more displays 412 correspond to diffractive,reflective, polarized, holographic, etc. waveguide displays. In oneexample, the device 10 includes a single display. In another example,the device 10 includes a display for each eye of the user 25, e.g., anHMD. In some implementations, the one or more displays 412 are capableof presenting MR content, including VR or AR content.

In some implementations, the one or more image sensor systems 414 areconfigured to obtain image data that corresponds to at least a portionof the face of the user 25 that includes the eyes of the user 25. Forexample, the one or more image sensor systems 414 include one or moreRGB camera (e.g., with a complimentary metal-oxide-semiconductor (CMOS)image sensor or a charge-coupled device (CCD) image sensor), monochromecamera, IR camera, event-based camera, and/or the like. In variousimplementations, the one or more image sensor systems 414 furtherinclude illumination sources that emit light upon the portion of theface of the user 25, such as a flash or a glint source.

The memory 420 includes high-speed random-access memory, such as DRAM,SRAM, DDR RAM, or other random-access solid-state memory devices. Insome implementations, the memory 420 includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid-state storagedevices. The memory 320 optionally includes one or more storage devicesremotely located from the one or more processing units 302. The memory420 comprises a non-transitory computer readable storage medium. In someimplementations, the memory 420 or the non-transitory computer readablestorage medium of the memory 420 stores the following programs, modulesand data structures, or a subset thereof including an optional operatingsystem 430 and a user experience module 440.

The operating system 430 includes procedures for handling various basicsystem services and for performing hardware dependent tasks. In someimplementations, the user experience module 440 is configured to presenta user experience that utilizes physiological data 45 to identify aninterest or intent 40 of the user 25 via context aware dynamicdistortion correction to the user 25 via the one or more input/output(I/O) devices and sensors 406. To that end, in various implementations,the user experience module 440 includes a physiological characteristictracking unit 442, an interest or intention unit 444, and a presentingunit 446.

In some implementations, the physiological characteristic tracking unit442 is configured to obtain physiological data (e.g., pupil dilation,electroencephalography, etc.) and to use the obtained physiological datato identify patterns of physiological data. To that end, in variousimplementations, the physiological characteristic tracking unit 442includes instructions and/or logic therefor, and heuristics and metadatatherefor.

In some implementations, the interest or intention unit 444 isconfigured to use the identified patterns of physiological data toidentify the interest or intent of a user of the device. To that end, invarious implementations, the interest or intention unit 444 includesinstructions and/or logic therefor, and heuristics and metadatatherefor.

In some implementations, the presenting unit 446 is configured topresent content via the one or more displays 412 based on the identifiedinterest or intent. To that end, in various implementations, thepresenting unit 446 includes instructions and/or logic therefor, andheuristics and metadata therefor.

Although the physiological characteristic tracking unit 442, interest orintention unit 444, and presenting unit 448 are shown as residing on asingle device (e.g., the device 10), it should be understood that inother implementations, any combination of these units may be located inseparate computing devices.

Moreover, FIG. 4 is intended more as functional description of thevarious features which are present in a particular implementation asopposed to a structural schematic of the implementations describedherein. As recognized by those of ordinary skill in the art, items shownseparately could be combined and some items could be separated. Forexample, some functional modules shown separately in FIG. 4 could beimplemented in a single module and the various functions of singlefunctional blocks could be implemented by one or more functional blocksin various implementations. The actual number of modules and thedivision of particular functions and how features are allocated amongthem will vary from one implementation to another and, in someimplementations, depends in part on the particular combination ofhardware, software, and/or firmware chosen for a particularimplementation.

FIG. 5 illustrates a block diagram of an exemplary head-mounted device500 in accordance with some implementations. The head-mounted device 500includes a housing 501 (or enclosure) that houses various components ofthe head-mounted device 500. The housing 501 includes (or is coupled to)an eye pad (not shown) disposed at a proximal (to the user 25) end ofthe housing 501. In various implementations, the eye pad is a plastic orrubber piece that comfortably and snugly keeps the head-mounted device500 in the proper position on the face of the user 25 (e.g., surroundingthe eye of the user 25).

The housing 501 houses a display 510 that displays an image, emittinglight towards or onto the eye of a user 25. In various implementations,the display 510 emits the light through an eyepiece having one or morelenses 505 that refracts the light emitted by the display 510, makingthe display appear to the user 25 to be at a virtual distance fartherthan the actual distance from the eye to the display 510. For the user25 to be able to focus on the display 510, in various implementations,the virtual distance is at least greater than a minimum focal distanceof the eye (e.g., 7 cm). Further, in order to provide a better userexperience, in various implementations, the virtual distance is greaterthan 1 meter.

The housing 501 also houses a tracking system including one or morelight sources 522, camera 524, and a controller 580. The one or morelight sources 522 emit light onto the eye of the user 25 that reflectsas a light pattern (e.g., a circle of glints) that can be detected bythe camera 524. Based on the light pattern, the controller 580 candetermine an eye tracking characteristic of the user 25. For example,the controller 580 can determine a gaze direction and/or a blinkingstate (eyes open or eyes closed) of the user 25. As another example, thecontroller 580 can determine a pupil center, a pupil size, or a point ofregard. Thus, in various implementations, the light is emitted by theone or more light sources 522, reflects off the eye of the user 25, andis detected by the camera 524. In various implementations, the lightfrom the eye of the user 25 is reflected off a hot mirror or passedthrough an eyepiece before reaching the camera 524.

The display 510 emits light in a first wavelength range and the one ormore light sources 522 emit light in a second wavelength range.Similarly, the camera 524 detects light in the second wavelength range.In various implementations, the first wavelength range is a visiblewavelength range (e.g., a wavelength range within the visible spectrumof approximately 400-700 nm) and the second wavelength range is anear-infrared wavelength range (e.g., a wavelength range within thenear-infrared spectrum of approximately 700-1400 nm).

In various implementations, eye tracking (or, in particular, adetermined gaze direction) is used to enable user interaction (e.g., theuser 25 selects an option on the display 510 by looking at it), providefoveated rendering (e.g., present a higher resolution in an area of thedisplay 510 the user 25 is looking at and a lower resolution elsewhereon the display 510), or correct distortions (e.g., for images to beprovided on the display 510).

In various implementations, the one or more light sources 522 emit lighttowards the eye of the user 25 which reflects in the form of a pluralityof glints.

In various implementations, the camera 524 is a frame/shutter-basedcamera that, at a particular point in time or multiple points in time ata frame rate, generates an image of the eye of the user 25. Each imageincludes a matrix of pixel values corresponding to pixels of the imagewhich correspond to locations of a matrix of light sensors of thecamera. In implementations, each image is used to measure or track pupildilation by measuring a change of the pixel intensities associated withone or both of a user's pupils.

In various implementations, the camera 524 is an event camera comprisinga plurality of light sensors (e.g., a matrix of light sensors) at aplurality of respective locations that, in response to a particularlight sensor detecting a change in intensity of light, generates anevent message indicating a particular location of the particular lightsensor.

It will be appreciated that the implementations described above arecited by way of example, and that the present invention is not limitedto what has been particularly shown and described hereinabove. Rather,the scope includes both combinations and sub combinations of the variousfeatures described hereinabove, as well as variations and modificationsthereof which would occur to persons skilled in the art upon reading theforegoing description and which are not disclosed in the prior art.

FIG. 6, in accordance with some implementations, is a flowchartrepresentation of a method 600 for initiating a user interaction basedon an identified interest or an identified intention of a user where theinterest or the intention is identified based on physiological data ofthe user obtained using a sensor during a user experience. In someimplementations, the method 600 is performed by one or more devices. Themethod 600 can be performed at a mobile device, HMD, desktop, laptop, orserver device. The method 600 can be performed on an HMD that has ascreen for displaying 3D images or a screen for viewing stereoscopicimages. In some implementations, the method 600 is performed byprocessing logic, including hardware, firmware, software, or acombination thereof. In some implementations, the method 600 isperformed by a processor executing code stored in a non-transitorycomputer-readable medium (e.g., a memory).

At block 610, the method 600 obtains physiological data at a device of auser during a user experience in which content is displayed on a displayof the device. In some implementations, the physiological data includeselectroencephalography (EEG), electrocardiography (ECG),electromyography (EMG), functional near-infrared spectroscopy (fNIRS),blood pressure, skin conductance, pupillary response, or any combinationthereof. For example, physiological data may be pupillary response,where the diameter of the pupil is measured over a period of time.

In some implementations, the method 600 performs a training function bypresenting the user with content and instructions directing the user'sinterest or intent. The method 600 may record the physiological dataassociated with the presentation of the content in order to identify apattern associated with the timing of the instructed moment of intent orinterest.

At block 620, the method 600 detects a pattern using the physiologicaldata. In some implementations, the method 600 compares the physiologicaldata obtained at block 610 to a pattern associated with user interest orintent. In some implementations, the method 600 accounts for anyexogenous signals that may affect detecting a pattern. For example,increased ambient light may result in an exogenous signal affectingpupillary response.

At block 630, the method 600 identifies an interest of the user in thecontent or an intention of the user regarding the content based ondetecting the pattern. In some implementations, as the user interactswith the device, the device detects the pattern associated with step 620in order to identify a current interest or intent of the user. In someimplementations, the current interest on intent is a selectable objectdisplayed on the screen of the device. Moreover, the interest or intentmay be identified based on the identified pattern in combination withanother input, e.g., confirming an intention of a hand gesture byidentifying a pattern associated with physiological data.

At block 640, the method 600 initiates a user interaction based on theidentified interest or the identified intention. In someimplementations, the user interaction includes moving an objectselection indicator, selecting an object, changing a state of aninteractive item, making a record of the object of interest or intent,or otherwise performing an operation associated with a given interactiveitem. For example, confidence in a user's current interest level orintention to interact may result in a discrete user interaction (e.g., a“click”-like event) or continuous user action spanning a period of time(e.g., a lingering event).

The present disclosure contemplates that users will be provided with anoption to benefit from any use of physiological data 45 to identify userintention or interest, improve a user interface, or otherwise improve auser experience. For example, a user may tailor device preferences toselect whether or not to use physiological data to enhance a userexperience. Moreover, a user may be provided with an option to selectactions or types of actions that may or may not be invoked automaticallybased on any use of physiological data 45.

The present disclosure further contemplates that the entitiesresponsible for the collection, analysis, disclosure, transfer, storage,or other use of such personal information and/or physiological data 45will comply with well-established privacy policies and/or privacypractices. In particular, such entities should implement andconsistently use privacy policies and practices that are generallyrecognized as meeting or exceeding industry or governmental requirementsfor maintaining personal information data private and secure. Forexample, personal information from users should be collected forlegitimate and reasonable uses of the entity and not shared or soldoutside of those legitimate uses. Further, such collection should occuronly after receiving the informed consent of the users. Additionally,such entities would take any needed steps for safeguarding and securingaccess to such personal information data and ensuring that others withaccess to the personal information data adhere to their privacy policiesand procedures. Further, such entities can subject themselves toevaluation by third parties to certify their adherence to widelyaccepted privacy policies and practices.

In the case of advertisement delivery services, the present disclosurealso contemplates scenarios in which users selectively block the use of,or access to, personal information data and/or physiological data 45.That is, the present disclosure contemplates that hardware and/orsoftware elements can be provided to prevent or block access to suchpersonal information data. For example, in the case of advertisementdelivery services, the present technology can be configured to allowusers to select to “opt in” or “opt out” of participation in thecollection of personal information data during registration forservices.

Therefore, although the present disclosure broadly covers use ofpersonal information data including physiological data 45 to implementone or more various disclosed implementations, the present disclosurealso contemplates that the various implementations can also beimplemented without the need for accessing such personal informationdata. That is, the various implementations of the present technology arenot rendered inoperable due to the lack of all or a portion of suchpersonal information data. For example, content can be selected anddelivered to users by inferring preferences based on non-personalinformation data or a bare minimum amount of personal information, suchas the content being requested by the device 10 associated with a user25, other non-personal information available to the content deliveryservices, or publicly available information.

Numerous specific details are set forth herein to provide a thoroughunderstanding of the claimed subject matter. However, those skilled inthe art will understand that the claimed subject matter may be practicedwithout these specific details. In other instances, methods apparatuses,or systems that would be known by one of ordinary skill have not beendescribed in detail so as not to obscure claimed subject matter.

Unless specifically stated otherwise, it is appreciated that throughoutthis specification discussions utilizing the terms such as “processing,”“computing,” “calculating,” “determining,” and “identifying” or the likerefer to actions or processes of a computing device, such as one or morecomputers or a similar electronic computing device or devices, thatmanipulate or transform data represented as physical electronic ormagnetic quantities within memories, registers, or other informationstorage devices, transmission devices, or display devices of thecomputing platform.

The system or systems discussed herein are not limited to any particularhardware architecture or configuration. A computing device can includeany suitable arrangement of components that provides a resultconditioned on one or more inputs. Suitable computing devices includemultipurpose microprocessor-based computer systems accessing storedsoftware that programs or configures the computing system from ageneral-purpose computing apparatus to a specialized computing apparatusimplementing one or more implementations of the present subject matter.Any suitable programming, scripting, or other type of language orcombinations of languages may be used to implement the teachingscontained herein in software to be used in programming or configuring acomputing device.

Implementations of the methods disclosed herein may be performed in theoperation of such computing devices. The order of the blocks presentedin the examples above can be varied for example, blocks can bere-ordered, combined, or broken into sub-blocks. Certain blocks orprocesses can be performed in parallel.

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor value beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

It will also be understood that, although the terms “first,” “second,”etc. may be used herein to describe various elements, these elementsshould not be limited by these terms. These terms are only used todistinguish one element from another. For example, a first node could betermed a second node, and, similarly, a second node could be termed afirst node, which changing the meaning of the description, so long asall occurrences of the “first node” are renamed consistently and alloccurrences of the “second node” are renamed consistently. The firstnode and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particularimplementations only and is not intended to be limiting of the claims.As used in the description of the implementations and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “comprises” or “comprising,” when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in accordance with a determination”or “in response to detecting,” that a stated condition precedent istrue, depending on the context. Similarly, the phrase “if it isdetermined [that a stated condition precedent is true]” or “if [a statedcondition precedent is true]” or “when [a stated condition precedent istrue]” may be construed to mean “upon determining” or “in response todetermining” or “in accordance with a determination” or “upon detecting”or “in response to detecting” that the stated condition precedent istrue, depending on the context.

The foregoing description and summary are to be understood as being inevery respect illustrative and exemplary, but not restrictive, and thescope of the invention disclosed herein is not to be determined onlyfrom the detailed description of illustrative implementations butaccording to the full breadth permitted by patent laws. It is to beunderstood that the implementations shown and described herein are onlyillustrative of the principles of the present invention and that variousmodification may be implemented by those skilled in the art withoutdeparting from the scope and spirit of the invention.

What is claimed is:
 1. A method comprising: at a device comprising aprocessor, a computer-readable storage medium, a display, and a sensor:obtaining physiological data of a user while content is displayed on thedisplay, the physiological data obtained using the sensor and varyingover time during the display of the content; based on detecting thephysiological data, identifying an interest or an intention of the userto interact with an interactive item in the content, wherein theinteractive item comprises a menu item, a displayed item capable ofchanging state, or a displayed item capable of being selected by aninput device; and initiating a user interaction with the interactiveitem based on identifying the interest or intention.
 2. The method ofclaim 1, wherein: the physiological data is pupil dilation datarepresenting a time-varying pupil diameter; and detecting the patterncomprises detecting a pupil dilation pattern.
 3. The method of claim 2,wherein detecting the pupil dilation pattern comprises accounting forexogenous signals corresponding to pupil diameter changes in the pupildilation data resulting from ambient light changes, chromatic changes,accommodation of the eye, content lighting changes, cyclical pupildilations, a change in ambient noise, or change in motion of the device.4. The method of claim 1, wherein detecting the pattern comprisesapplying a machine learning technique trained to identify patterns inphysiological data corresponding to user interests or user intentions.5. The method of claim 1, wherein the physiological data representsinvoluntary data.
 6. The method of claim 1 further comprising obtainingvoluntary data, wherein identifying the interest or intention is furtherbased on the voluntary data.
 7. The method of claim 6, wherein thevoluntary data comprises: a gesture of a body part detected by an imagesensor during the user experience; a voice command of a voice detectedby a sound sensor during the user experience; a fixed gaze detected byan eye sensor during the user experience; a sequence of gaze patternsdetected by an eye sensor during the user experience; an orientingresponse [e.g., head movement]; a movement detected by a motion sensorduring the user experience; a facial expression detected by an imagesensor during the user experience; or an attribute included in thecontent.
 8. The method of claim 1 further comprising progressivelyidentifying related interests or intentions based on previouslyidentified interests or previously identified intentions during the userexperience.
 9. The method of claim 1 further comprising determining aconfidence in the identified interest or the identified intention basedon previously identified interests or previously identified intentionsduring the user experience.
 10. The method of claim 1, whereinidentifying the intention comprises identifying an intent to interactwith the menu item.
 11. The method of claim 1, wherein identifying theintention comprises identifying an intent to interact with the displayeditem capable of changing state to change the state of the displayeditem.
 12. The method of claim 1, wherein identifying the intentioncomprises identifying an intent to select the displayed item capable ofbeing selected by an input device.
 13. The method of claim 1, whereinidentifying the intention comprises: receiving data regarding avoluntary user movement; and interpreting the voluntary user movement asan intention to interact with the interactive item based on involuntarydata in the physiological data.
 14. The method of claim 1, whereininitiating the user interaction comprises: providing additional contentassociation with an object in the content corresponding to theidentified interest or the identified intention; removing an object inthe content based on the identified interest or the identifiedintention; or automatically capturing images of the content at timesduring the user experience determined based on the identified interestor the identified intention.
 15. The method of claim 1, whereindetecting the pattern comprises: tracking a physiological attributeassociated with the physiological data using a first sensor; andactivating a second sensor to obtain the physiological data based on thetracking.
 16. The method of claim 1, wherein the physiological datacomprises electroencephalography (EEG) data of functional near infraredspectroscopy signal (fNIRS).
 17. The method of claim 1, furthercomprising receiving informed consent of the user to obtain thephysiological data or voluntary data of the user.
 18. A systemcomprising: an electronic device with a display and a sensor; aprocessor; and a computer-readable storage medium comprisinginstructions that upon execution by the processor cause the system toperform operations, the operations comprising: obtaining physiologicaldata of a user while content is displayed on the display, thephysiological data obtained using the sensor and varying over timeduring the display of the content; based on detecting the physiologicaldata, identifying an interest or an intention of the user to interactwith an interactive item in the content, wherein the interactive itemcomprises a menu item, a displayed item capable of changing state, or adisplayed item capable of being selected by an input device; andinitiating a user interaction with the interactive item based onidentifying the interest or intention.
 19. The system of claim 18,wherein: the physiological data is pupil dilation data representing atime-varying pupil diameter; and detecting the pattern comprisesdetecting a pupil dilation pattern, wherein detecting the pupil dilationpattern comprises accounting for exogenous signals corresponding topupil diameter changes in the pupil dilation data resulting from ambientlight changes, chromatic changes, accommodation of the eye, contentlighting changes, cyclical pupil dilations, a change in ambient noise,or change in motion of the device.
 20. A non-transitorycomputer-readable storage medium storing program instructions that arecomputer-executable to perform operations comprising: obtainingphysiological data of a user while content is displayed on the display,the physiological data obtained using the sensor and varying over timeduring the display of the content; based on detecting the physiologicaldata, identifying an interest or an intention of the user to interactwith an interactive item in the content, wherein the interactive itemcomprises a menu item, a displayed item capable of changing state, or adisplayed item capable of being selected by an input device; andinitiating a user interaction with the interactive item based onidentifying the interest or intention.